completed should be 0, since you didn't read anything. So here you could also return Ok(0) ("no more data") and let the if below return Error::PartialBuffer.

Right!

Same as above.

Indeed.

What about defining this like:

pub struct Permissions(u8);
impl Permissions {
    pub const No: Permissions = Permissions(0);
    pub const Read: Permissions = Permissions(1);
    pub const Write: Permissions = Permissions(2);
    pub const ReadWrite: Permissions = Permissions(3);
}

and implementing & and | as just a bitwise operation on x.0? (Basically reinventing the bitflags crate to avoid introducing a new dependency)

(and also reinventing it to hide the underlying implementation of the fake enum)

I’m not dead-set; for bitfields, such a struct with constants can make more sense. On the other hand, I like using an enum if anything just to check that match is exhaustive (e.g. when we need to translate Permissions into VhostAccess).

FWIW, starting from opt-level = 2, this does get optimized properly:

   0:   48 89 f8                mov    rax,rdi
   3:   48 21 f0                and    rax,rsi
   6:   c3                      ret

As a compromise, I could do repr(u8) with bitand(x, y) { ((x as u8) | (y as u8)).try_into().unwrap() and an accompanying TryFrom<u8> implementation. Even on opt-level = 0, that would yield:

   0:   48 89 f8                mov    rax,rdi
   3:   48 89 44 24 f0          mov    QWORD PTR [rsp-0x10],rax
   8:   48 89 74 24 f8          mov    QWORD PTR [rsp-0x8],rsi
   d:   48 21 f0                and    rax,rsi
  10:   c3                      ret

(opt-level = 1 then also removes the stack accesses, as above)

On the other hand, I like using an enum if anything just to check that match is exhaustive (e.g. when we need to translate Permissions into VhostAccess).

Ah, you're right. Exhaustiveness is important.

Let's go for the repr, but with a private panicking fn from_repr(x: u8 -> Self) instead of the TryFrom trait.

Sounds good!

I'd consider just changing GuestMemory. Is anyone using try_access directly? It's not private, but almost so.

I did consider that, and I put the reasoning why I decided not to do so (and indeed failed to do so) into the opening comment: The most immediate reason is that anyone accessing memory needs to specify the required access permissions. Yes, most accesses go through the Bytes interface, i.e. already specify the intended access, but not all of them:

Which gets me to the second question: There are users (like descriptor_utils used by virtiofsd or vm-virtio/virtio-queue) that currently use get_slice(). get_slice() is dangerous because it cannot cope with fragmented memory, which becomes especially apparent when using paged virtual memory. So in fact my current draft replaces those get_slice() usages by try_access().

We could of course replace get_slice() by something like get_slices() -> impl Iterator<Item = VolatileSlice>, but really, that wouldn’t offer more than the try_access() we already have, just replace the interior iterator by an exterior one. (Though I’m open to the idea.)

Besides the access mode, there’s also the general idea I outlined in the original comment that GuestMemory is quite open about how it is structured into a limited set of GuestMemoryRegion objects internally. For virtual memory, that’s no longer applicable: Pages and guest memory regions are not only conceptually different, access to single pages is also quite useless. (Whereas GuestMemoryRegion is fundamental to how the vhost code constructs the GuestMemory from the information it gets from the front-end, so it must be visible.)

If we continued to use GuestMemory, I’m afraid it will be impossible to verify at compile time that all the accesses are sound. For example, a user could try to access virtual memory’s GuestMemoryRegion objects, and that would only return an error at runtime.

Yes, that's pretty convincing. The one thing I'd change however is to always expose IoMemory and only put IommuMemory/Iommu/Iotlb under the new feature.

This way, the Bytes<GuestAddress> implementation can simply move to IoMemory.

That’s how it already is, though. I’m happy to keep it that way 😅 but by putting IoMemory under the iommu feature, users could remain unchanged until they need/want IOMMU support.

(The problem is that the blanket impl Bytes<_> for IoMemory plus blanket impl IoMemory for GuestMemory does not auto-extend to Bytes<_> for GuestMemory, so with the introduction of IoMemory, users need to be changed to at least import IoMemory.)

I think you'd get it just by importing Bytes...

The problem with moving IoMemory to the new feature is that features are additive. You can't make the iommu feature pick whether Bytes<> is added to GuestMemory or IoMemory.

Unrelated to this, maybe we should bite the bait and add a prelude...

Testcase:

lib.rs:

pub trait GuestMemory {}
pub trait IoMemory {}

impl<T: GuestMemory> IoMemory for T {}

pub trait HelloWorld {
    fn hello_world(&self);
}

impl<T: IoMemory> HelloWorld for T {
     fn hello_world(&self) {}
}

test.rs:

use lib::{GuestMemory, HelloWorld};

pub struct Foo;
impl GuestMemory for Foo {}

fn main() {
    Foo.hello_world();
}

$ rustc --crate-type=rlib lib.rs --edition=2021
$ rustc test.rs --extern lib=liblib.rlib --edition=2021

Interesting. It doesn’t work for vm-virtio/virtio-queue because that has a relaxed ?Sized requirement. Extending your example:

use lib::{GuestMemory, HelloWorld};

pub struct Foo;
impl GuestMemory for Foo {}

pub fn hello<T>(foo: &T)
where
    T: GuestMemory + ?Sized,
{
    foo.hello_world();
}

fn main() {
    hello(&Foo);
}

It works fine without the ?Sized.

Removing the ?Sized from virtio-queue and adding + Sized to <M as Deref>::Target: GuestMemory where that appears makes it work indeed. But I’m not sure whether they’ll be happy about that…

Great news! If I just add + ?Sized to the IoMemory for GuestMemory impl, it works:

pub trait IoMemory {
    type PhysicalMemory: GuestMemory + ?Sized;
    /* ... */
}

impl<M: GuestMemory + ?Sized> IoMemory for M {
    /* ... */
}

I am not sure I like the design here. The IOVA address space, being virtual, can have aliases. Could you still store the bitmap in the low-level memory region, but with accesses going through IOMMU translation?

What does vhost-user require? Are dirty bitmap accesses done in IOVA or GPA space?

Are dirty bitmap accesses done in IOVA or GPA space?

They’re done in IOVA space, which is why I think we need the bitmap on this level.

(If we continued to store it in e.g. GuestRegionMmap/MmapRegion, it would need a reverse translation, and then keep a mapping of address ranges to bitmap slices instead of just a single bitmap slice linearly covering the entire region.)

They’re done in IOVA space, which is why I think we need the bitmap on this level.

I see. Maybe you could add a bitmap::BS<'a, Self::Bitmap> as another argument that try_access() passes back to the callback? And then IommuMemory::get_slice() can pass that argument to replace_bitmap().

This way, the Iommu controls whether dirty tracking is done in IOVA or GPA space.

You mean putting the responsibility on the try_access() callback to dirty a bitmap slice given to it if it has written anything?

I’m not sure, that does sound more than reasonable, but would require more changes in the callers than just to add the Permissions flag. 🙂

You mean putting the responsibility on the try_access() callback to dirty a bitmap slice given to it if it has written anything?

No, I confused try_access and get_slice sorry. For IoMemory, dirtying is already done it in try_access itself; for Iommu the slice could be returned by translate, that is it would be part of the Iotlb entry?

I’m sorry, I don’t quite follow; currently, dirtying is done only by IommuMemory (in its IoMemory implementation), neither IoMemory nor Iommu.

<IommuMemory as IoMemory>::try_access() dirties the bitmap itself, right.

<IommuMemory as IoMemory>::get_slice() has the VolatileSlice do the work. For this, it replaces the VolatileSlice’s internal bitmap slice (which is initially set by the GuestMemoryRegion) by the right slice in the IOVA space.

I don’t think the IOMMU object should do any of this, and I don’t think it should return slices (if I understand correctly). I think it should only do the translation and not care about the actual memory.